Gradual transitioning between two-dimensional and three-dimensional augmented reality images

ABSTRACT

System and method for enhancing situational awareness. A moveable see-through display viewable by a user displays an augmented reality 2D image of an external scene based on received 2D image data, in accordance with updated position and orientation of display. The see-through display further displays an augmented reality 3D image of the external scene based on received 3D image data, the 3D image overlaid conformally onto view of external scene, in accordance with updated position and orientation of display. The see-through display further selectively displays: a gradual transition of the 2D image into the 3D image, or a gradual transition of the 3D image into the 2D image. At least one image feature may gradually appear or gradually disappear during the gradual transition. The 2D or 3D image may include a region of interest based on updated position and orientation of display or selected by user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a National Phase application filed under 35 U.S.C. §371 of PCT International Application No. PCT/IL2018/050252 with anInternational Filing Date of Mar. 6, 2018, which claims priority toIsrael Patent Application No. 251189, filed on Mar. 15, 2017, the entirecontents of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to see-through displays,synthetic vision systems, digital maps, and three-dimensionalvisualization.

BACKGROUND OF THE INVENTION

A synthetic vision system (SVS) is a form of mediated reality thatdisplays a synthetic image of the external environment in order toprovide the viewer with an improved understanding of his/hersurroundings. An SVS is commonly employed in aircrafts to provide thepilot or flight crew member with a visual representation of the aircraftflight path with respect to the external environment. The syntheticimagery generally depicts environmental features, such as terrain orobstacles encountered along the flight trajectory, together with symbolsor indicators representing relevant flight parameters such as: airspeed;altitude; heading; a horizon line; turn/bank/slip/skid parameters; andthe like. The information may be obtained from maps and databases storedonboard the aircraft, as well as aircraft sensors and navigationalsystems. Synthetic vision may also be employed in non-aerial platforms,such as automobiles or other land vehicles, in order to assist drivingor navigation during nighttime or poor visibility weather conditions. AnSVS may also be combined with an enhanced vision system (EVS), whichdisplays real-time images of the environment obtained from supplementarycameras or sensors, such as thermal cameras or radar detectors.

An SVS typically includes at least an image generator for generating thesynthetic image, and a display for displaying the synthetic image. Forexample, the synthetic image may be projected onto a fixed display inthe aircraft cockpit, such as a head-up display (HUD) or a head-downdisplay (HDD). A HUD may be transparent or “see-through”, enabling thepilot to view the synthetic image while maintaining a forward viewpointof the physical environment in the background, avoiding the need todivert attention elsewhere to a separate display or instrumentationpanel. The synthetic image may also be projected onto a movable displaydevice that moves in conjunction with the head of the pilot, known as ahead-mounted display (HMD), which may also provide a simultaneoussee-through view of the external environment.

An SVS presents the synthetic image in a three-dimensional (3D) formatso as to provide a perspective view of the environment. Othervisualization systems may display two-dimensional (2D) images of theoutside world, such as a 2D digital map extracted from a digital terrainmodel (DTM). A 2D image generally allows the viewer to easily identifydifferent features in the image, such as representations of nearbylandmarks or objects, but it may be difficult to comprehend where thesefeatures are situated in relation to the physical surroundings, astranslating a two-dimensional image representation into a correspondingthree-dimensional real-world object is not intuitive or straightforward.For example, the pilot may view a 2D road map of a geographic regionprojected onto the real-world view of the same geographic region seenfrom above. The 2D map provides the pilot with an understanding of thedifferent terrain features seen on the map, but does not necessarilyprovide a precise indication of how exactly these features relate to thecurrent location of the pilot, in terms of the real-time position andorientation of the aircraft. For example, a particular geographicfeature viewed on the 2D map and indicated as being a point of interest(POI) may be positioned adjacent to a second feature with a similarappearance and attributes, such that it may be unclear or ambiguouswhich of the two features represents the indicated POI. Such confusionmay occur between visually similar elements that are located relativelyclose together in the 2D image, such as a symbolic indication referringto a specific obstacle among multiple obstacles in the aircraft flightpath. In certain situations, particularly with aircraft piloting, anindication in the displayed synthetic image may be time-sensitive orotherwise of crucial importance, where adverse consequences may resultif the displayed information is incorrectly perceived or improperlycomprehended by the viewer. Moreover, there may be certain geographicfeatures of interest that are partially or entirely obstructed from viewto the pilot due to the particular position and orientation of theaircraft, and thus the pilot is unaware of these obstructed features inhis surroundings. For example, a POI may be positioned behind a mountainridge. Such a POI would not be noticeable in a 3D map image (SVS) or inthe real world, but would be noticeable in a 2D map image.

U.S. Pat. No. 6,229,546 to Lancaster et al, entitled: “Rapid terrainmodel generation with 3-D object features and user customizationinterface”, discloses a method and system for generating a 3D worldmodel for simulated real terrain optimized for a personal computer.Terrain and other environmental data is acquired from digital datasources and processed to construct a predetermined intermediate databaseformat. The data is extracted and processed to create a 3D world modelfile in a format optimized for a particular imaging display software,such as browsers compliant with the Virtual Reality Modeling Languagespecification. In the formatting step, the simulated land surface iscolored and textured to correspond to geographic database layers, andnatural and man-made structures are made to populate the terrain skin as3D objects.

U.S. Pat. No. 7,352,292 to Alter et al, entitled: “Real-time,three-dimensional synthetic vision display of sensor-validated terraindata”, is directed to a synthetic vision system that provides asynthetic view of terrain and obstacles in the vicinity of a movingvehicle. The synthetic view is a computer rendering in 3D perspective ofobjects in a terrain database. The database is updated in real-time inresponse to data obtained by a ranging sensor. The updated database maybe stored and shared with users of displays in other vehicles.

U.S. Pat. No. 7,856,370 to Katta et al, entitled: “Method and system fordisplaying predictions on a spatial map”, discloses a method and systemfor making and displaying predictions on a spatial map. A data analyzeranalyzes heterogeneous data having spatial components to find utilizabledata, and uses machine learning and other methods to extractrelationships from the utilizable data. The extracted relationships areused to make a prediction about at least one location on the spatialmap, or to compare numerous locations. An interface presents theprediction on the spatial map in the form of a heat map overlying a 3Dtopographical map. The 3D map may be shown as an oblique or orthogonalprojection, or a perspective view. The heat map may be 2D or 3D andselectively displayed depending on user preference.

U.S. Pat. No. 8,264,498 to VanDerKamp et al., entitled: “System,apparatus, and method for presenting a monochrome image of a terrain ona head-up display unit”, discloses a system, apparatus and method forpresenting a monochrome 3D terrain image to a pilot on a HUD. Aircraftnavigation and terrain data is received, and an image data setrepresentative of a non-wire-frame single color perspective of theterrain scene outside the aircraft is generated, as a function ofterrain data and color intensity data. The color intensity data mayinclude shading effects and/or texturing effects. The shading effectsare such that changes in terrain elevation or terrain contours areindicated by varied brightness of a single color, where darker andlighter areas of the 3D terrain image correlate to greater and lessertransparencies of the HUD.

U.S. Pat. No. 8,400,330 to He et al., entitled: “System for displayingmultiple overlaid images to a pilot of an aircraft during flight”, isdirected to the displaying of multiple images to a pilot. A sensorsubsystem detects a light transmission originating outside the aircraftand generates a first signal indicative of the light transmission. Adynamic condition sensor detects a dynamic condition of the aircraft andgenerates a second signal indicative of the dynamic condition. Aprocessor commands a display unit to display a first image correspondingto the first signal, and a second image, overlaid over the first image,corresponding to the second signal, and to modify the appearance of thesecond image to enhance the ability of the pilot to discern the firstimage. For example, the processor may command the display unit to blankout a portion of the second image, to render a portion of the secondimage partially transparent, or to diminish a brightness of a portion ofthe second image.

U.S. Pat. No. 8,687,056 to Yahav et al, entitled: “Aircraft landingassistance”, discloses an enhanced vision system for assisting aircraftpiloting. An aircraft control operator sends flight instructionsassociated with an object of interest to a pilot wearing a head-mounteddisplay (HMD). A visual representation of the flight instructions withthe object of interest marked is generated, respective of a combinedspatial and symbolic image viewed by the pilot on the HMD. The aircraftcontrol operator receives from the pilot confirmation of the flightinstructions by designating the marked object of interest on thecombined spatial and symbolic image, where the designation is performedin conjunction with the pilot line-of-sight.

U.S. Patent Application No. 2011/0052042 to Ben Tzvi, entitled:“Projecting location based elements over a heads up display”, disclosesa method and system for displaying location aware entities (LAEs) over avehicle windshield while driving to provide navigation guidance. A 3Dmodel of a scene within a specified radius of the vehicle is generatedfrom a digital mapping source, and a position of a selected LAEcontained within the scene is associated with a respective position inthe 3D model. The LAE is superimposed onto a specified position on atransparent screen facing the viewer and associated with the vehicle,with a graphic indicator associated with the LAE. The specified positionis calculated based on: the respective position of the LAE in the 3Dmodel; the screen geometrical and optical properties; the viewing angle;the viewer distance from the screen; and/or the vehicle position andangle within the scene, such that the graphic indicator and LAE aresubstantially on a common line. The graphic indicator may be a directionarrow, which is repeatedly updated based on the vehicle position on acalculated route between the vehicle and LAE.

U.S. Patent Application No. 2014/0316611 to Da Silva, entitled: “Systemand method of operation for remotely operated vehicles with superimposed3D imagery”, is directed to the operation of remotely operated vehicles(ROVs) with superimposed 3D imagery and navigational information. A livevideo feed is acquired by a video camera of the ROV. A virtual videofeed incorporating 3D elements representing objects disposed in anoperation environment of the ROV is generated. The angle and position ofthe virtual video camera is synchronized with the angle and position ofthe real video camera. The virtual video feed and real video feed aresuperimposed, such that the transparency or opaqueness of a region oflesser interest in one video feed is manipulated to make thecorresponding region of the other video feed more visible. Graphicand/or textual information may also be superimposed onto the hybrid 3Dimagery.

U.S. Patent Application No. 2010/0225644 to Swope, III et al, entitled:“Method and system for transitioning between views in a traffic report”,is directed to a traffic report that includes a visual depiction of ageographical area and can transition between two types of views. Datarepresenting traffic conditions is received from various sources and isused by a traffic report application to generate a video output, such ason a web-based or cellular-based application, that depicts at least twotypes of geographic graphics. The traffic report moves from a first viewof a virtual world to a second view of the virtual world such that bothparts of the virtual world are visible for at least part of thetransition. The transition may be from a 2D view to a 3D view, from a 3Dview to a 2D view, or between two different 3D views. The transition mayfade out elements of the first view, such as by increasing transparencyof the elements, while moving to the second view. Transitioning betweenviews may include changing the altitude, direction or orientation of avirtual camera towards the second view.

U.S. Patent Application No. 2014/0071119 to Piemonte et al, entitled:“Displaying 3D objects in a 3D map presentation”, is directed to thedisplay of building representations and other 3D object representationson a map presentation of a map application. When the map presentation ismoved to display a new area, the 3D representations rise from a groundlevel to their full heights and transition from transparent to opaque atthe same time. Areas can be brought into view by a command to pan themap or to zoom in below a threshold level. Conversely, the 3Drepresentations may be removed by lowering the objects from their fullheight to ground level and fading out from opaque to transparent. Thebuilding representations may also be depicted in a 2D map presentation,in which the building are depicted as flat, but may be caused to fadeand rise if the map presentation transitions to a 3D view.

SUMMARY OF THE INVENTION

In accordance with one aspect of the present invention, there is thusprovided a system for enhancing the situational awareness of a user. Thesystem includes a moveable see-through display viewable by the user, anda processor. The processor is configured to receive an updated positionand orientation of the see-through display, to receive two-dimensional(2D) image data relating to an external scene, and to receivethree-dimensional (3D) image data relating to the external scene. Thesee-through display is configured to display an augmented reality 2Dimage of the external scene based on the received 2D image data, inaccordance with the updated position and orientation of the see-throughdisplay. The see-through display is further configured to display anaugmented reality 3D image of the external scene based on the received3D image data, the 3D image overlaid conformally onto a view of theexternal scene, in accordance with the updated position and orientationof the see-through display. The see-through display is furtherconfigured to selectively display a gradual transition of the 2D imageinto the 3D image and/or a gradual transition of the 3D image into the2D image. The 2D image data or the 3D image data may include: map data,geographical terrain features, textural data relating to geographicalterrain features, a 3D geographical model, hierarchical map information,previously captured sensor images, real-time sensor image, and/or avideo image. The 2D image or the 3D image may include at least oneregion of interest (ROI), which may be selected based on the updatedposition and orientation of the display or may be selected by the user.The 3D image may depict at least one ROI obstructed by features in theexternal scene. The 2D image or 3D image may be a video image. At leastone image feature of the 2D image or the 3D image may graduallydisappear or may gradually appear during the gradual transition. Thedisplay may further display supplementary image content during thegradual transition. At least one transitioning attribute of the gradualtransition may be selectively modified. The display may repeatedlyupdate the gradual transition in accordance with the updated positionand orientation of the display. The 3D image may be generated in advancebased on: a radius surrounding the updated position and orientation ofthe display; a heading vector of the display, a selected ROI, and/orother predefined information. The system may further include aline-of-sight detector, configured to detect the position andorientation of the display. The movable display may be on a movingplatform, such as an aircraft in flight. The display may be: a head-updisplay (HUD), a head-mounted display (HMD), a wearable display device,and/or a display screen of a computing device.

In accordance with another aspect of the present invention, there isthus provided a method for enhancing the situational awareness of auser. The method includes the procedures of: receiving an updatedposition and orientation of a movable see-through display viewable bythe user, receiving 2D image data relating to an external scene, andreceiving 3D image data relating to the external scene. The methodfurther includes the procedures of: displaying on the see-throughdisplay an augmented reality 2D image of the external scene based on thereceived 2D image data, in accordance with the updated position andorientation of the see-through display; and displaying on thesee-through display an augmented reality 3D image of the external scenebased on the received 3D image data, the 3D image overlaid conformallyonto a view of the external scene, in accordance with the updatedposition and orientation of the see-through display. The method furtherincludes the procedure of selectively displaying on the see-throughdisplay a gradual transition of the 2D image into the 3D image and/or agradual transition of the 3D image into the 2D image. The 2D image orthe 3D image may include at least one ROI, which may be selected basedon the updated position and orientation of the display or may beselected by the user. The 3D image may depict at least one ROIobstructed by features in the external scene. The 2D image or 3D imagemay be a video image. At least one image feature of the 2D image or the3D image may gradually disappear or may gradually appear during thegradual transition. At least one transitioning attribute of the gradualtransition may be selectively modified. The gradual transition may berepeatedly updated in accordance with the updated position andorientation of the display. The 3D image may be generated in advancebased on: a radius surrounding the updated position and orientation ofthe display; a heading vector of the display, a selected ROI, and/orother predefined information.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood and appreciated more fully fromthe following detailed description taken in conjunction with thedrawings in which:

FIG. 1 is a schematic illustration of a system for enhancing thesituational awareness of a user, constructed and operative in accordancewith an embodiment of the present invention;

FIG. 2A is an illustrative view of an exemplary displayedtwo-dimensional (2D) map image with indicated regions of interest,operative in accordance with an embodiment of the present invention;

FIG. 2B is an illustrative view of an exemplary displayedthree-dimensional (3D) map image with indicated regions of interestcorresponding to the 2D map image of FIG. 2A, operative in accordancewith an embodiment of the present invention;

FIG. 3 is a schematic illustration of an exemplary 2D image graduallytransitioning into a 3D image, operative in accordance with anembodiment of the present invention;

FIG. 4 is a schematic illustration of a viewer observing a 2D image withan indicated region of interest (ROI) gradually transitioning into a 3Dindication of the ROI displayed conformally onto a view of a real-worldscene, operative in accordance with an embodiment of the presentinvention; and

FIG. 5 is a block diagram of a method for enhancing the situationalawareness of a user, operative in accordance with an embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present invention overcomes the disadvantages of the prior art byproviding a system and method for enhancing the situational awareness ofa user viewing a movable see-through display, by selectively displayinga gradual transition of a two-dimensional (2D) augmented reality imageof the external scene into a three-dimensional (3D) augmented realityimage of the external scene. The transition may also be reversed suchthat a 3D augmented reality image of the scene presented on the movablesee-through display is modified gradually into a 2D augmented image ofthe scene. The gradual transitioning allows the viewer to visually trackimage features, such as regions of interest in the scene, transitioningbetween the 2D image and 3D image formats. In this manner, the viewermay intuitively perceive the depicted image features in relation to thereal-world environment.

Reference is now made to FIG. 1, which is a schematic illustration of asystem, generally referenced 110, for enhancing the situationalawareness of a user, constructed and operative in accordance with anembodiment of the present invention. System 110 includes a processor112, a see-through display 114, a user interface 116, an image sensor118, a database 120, and a line-of-sight (LOS) detector 122. Processor112 is communicatively coupled with display 114, with user interface116, with image sensor 118, with database 120, and with LOS detector122. System 110 is generally installed on a platform, referenced 100,although some components may reside at a different location and may beaccessible to processor 112 through a wireless communication link. Forexample, system 110 may be implemented (at least partially) on anaircraft, an automobile, a motorcycle, a ship or marine vessel, or othertypes of moving platforms. Alternatively, system 110 may be installed(at least partially) on a stationary platform, in which case see-throughdisplay 114 is a movable display, such as a head-mounted display (HMD)that moves in accordance with the head movements of the user. The term“user” herein refers to any person or group of persons operating thesystem or method of the present invention. For example, the user may bean aircraft pilot or automobile driver, where the system is installed(at least partially) in the cockpit of an aircraft or automobile with aviewable see-through display. For another example, the user may be apedestrian and the system may be integrated with a wearable see-throughdisplay worn by the user.

Display 114 displays an image, such as a 2D image or a 3D image,generated or obtained by processor 112. Display 114 is a transparent or“see-through” display device, such that the user can simultaneouslyobserve the displayed image overlaid in the foreground onto a backgroundview of the external environment viewable through the display. Display114 is also a movable display, either a moving display on a fixed(stationary) platform, and/or a fixed (or moving display) on a movingplatform. For example, display 114 may be embodied by a fixed display,such as a head-up display (HUD) or a head-down display (HDD) integratedin a vehicle platform 100. Alternatively, display 114 may be ahead-mounted display (HMD) embedded within a wearable apparatus worn bythe user, or a portable or hand-held display, such as a display screenof a mobile computing device. Display 114 may include a projectorconfigured to project an image onto a display screen viewable by theuser.

User interface 116 allows the user to control various parameters orsettings associated with the components of system 110. For example, userinterface 116 can allow the user to provide instructions or selectparameters associated with the displayed image. User interface 116 mayinclude a cursor or touch-screen menu interface, such as a graphicaluser interface, configured to enable manual input of instructions ordata. User interface 116 may also include communication devicesconfigured to provide voice communication, such as a microphone and anaudio speaker, as well as voice recognition capabilities to enableentering instructions or data by means of speech commands. Userinterface 116 may also enable the user to communicate with externalsources, such as with a remote supervisor.

Image sensor 118 captures images of a scene in a real-world environment.Image sensor 118 may be any type of sensor device capable of acquiringan image representation of the scene, including the acquisition of anyform of electromagnetic radiation at any range of wavelengths (includingvisible and non-visible wavelengths). For example, image sensor 118 maybe a forward looking infrared (FLIR) camera or a charge-coupled device(CCD) camera. Image sensor 118 may be mounted on platform 100, and maybe aligned toward the general direction in which the user is facing, soas to image a scene in the field of view (FOV) of the user. Image sensor118 is operative to acquire at least one image frame, such as a sequenceof consecutive image frames representing a video image, which may beconverted into an electronic signal for subsequent processing and/ortransmission.

Database 120 stores information relating to real-world environments,such as an environment in which system 110 is expected to be located.The environmental information may include a 3D geographic model thatincludes a three-dimensional representation of the Earth or of aparticular area, region or territory of interest. The 3D geographicmodel may include image and texture data relating to geographicalfeatures, including artificial features (e.g., buildings or monuments),such as the location coordinates of such features and different viewsthereof (e.g., acquired via satellite imagery or aerial photography,and/or street level cameras). The 3D model may also provide multiplevisual representations of the geographical terrain of a region ofinterest at different positions and viewing angles. Database 120 mayinclude a digital elevation map, weather or climate forecasts, anddriving routes or flight routes of platform 100. Database 120 may alsoinclude previously captured images and/or image data that allows for thereconstruction of synthetic images of the relevant scene. Database 120may also store supplementary image content associated with differentlandmarks or geographic locations, such as in the form of symbols, text,or other graphics, to be selectively displayed with relevant images.Database 120 may be located externally to platform 100 butcommunicatively coupled with system 110, such that database 120 maytransmit images to system 110 while platform 100 is in motion.

LOS detector 122 provides an indication of the position and orientationof display 114. LOS detector 122 may include one or more devices orinstruments configured to measure the position and the orientation orviewing angle of display 114 with respect to a reference coordinatesystem, such as: a global positioning system (GPS); a compass; aninertial navigation system (INS); an inertial measurement unit (IMU);motion sensors or rotational sensors (e.g., accelerometers, gyroscopes,magnetometers); a rangefinder; and the like. LOS detector 122 mayutilize the location of platform 100 on which display 114 is situatedwhen calculating the position and orientation of display 114. LOSdetector 122 may further utilize a driving route or flight route ofplatform 100 (e.g., using a GPS or other onboard instruments), as wellas other relevant real-time parameters of platform 100, such as velocityand acceleration, to allow the determination of an updated location overtime. If display 114 is embodied by a head-mounted display (HMD), LOSdetector 122 may include a head tracking device configured to determinethe real-time head direction of the user, and/or an eye tracking deviceconfigured to determine the real-time eye gaze direction of the user.

Processor 112 receives instructions and data from the components ofsystem 110. Processor 112 performs necessary image processing anddirects the projection of an augmented reality image on see-throughdisplay 114, as will be discussed further hereinbelow. The componentsand devices of system 110 may be based in hardware, software, orcombinations thereof. It is appreciated that the functionalityassociated with each of the devices or components of system 110 may bedistributed among multiple devices or components, which may reside at asingle location or at multiple locations. For example, the functionalityassociated with processor 112 may be distributed between multipleprocessing units (such as a dedicated image processor for the imageprocessing functions). Processor 112 may be part of a server or a remotecomputer system accessible over a communications medium or network, ormay be integrated with other components of system 110, such asincorporated with a computer associated with display 114. System 110 mayoptionally include and/or be associated with additional components notshown in FIG. 1, for enabling the implementation of the disclosedsubject matter. For example, system 110 may include a power supply (notshown) for providing power to various components, and may furtherinclude a memory or storage unit (not shown) for temporary storage ofimages or other data.

The term “image” as used herein may refer to a video image or aplurality of image frames presented in sequence. In accordance with anembodiment of the present invention, a video image may be displayed inreal-time and continuously updated to reflect the actual environmentrespective of the current location and heading of a moving platform 100.

The term “repeatedly” as used herein should be broadly construed toinclude any one or more of: “continuously”, “periodic repetition” and“non-periodic repetition”, where periodic repetition is characterized byconstant length intervals between repetitions and non-periodicrepetition is characterized by variable length intervals betweenrepetitions.

The term “region of interest (ROI)” as used herein may refer to one ormore points, features or areas, of any size, shape, or configuration, inan external scene, including a collection of points that represent aunified physical object or entity located in the scene (i.e., an “objectof interest”), or that represent a general environmental feature orgroup of features (and not necessarily a unified object), and includingpoints or features that are dynamic (e.g., in motion relative to theuser or system).

The operation of system 110 will now be described in general terms,followed by specific examples. Processor 112 receives an indication ofthe current position and orientation of display 114 from LOS detector122. The position and orientation of display 114 represents a viewpointof an external scene viewed by the user though the display 114.Processor 112 further receives 2D image data and 3D image data of theexternal scene. For example, processor 112 may receive real-time imagesof the scene captured by image sensor 118, or may obtain from database120 previously captured images or environmental information from whichimages of the scene may be generated. Processor 112 generates (orreceives) a 2D image based on the 2D image data of the external sceneand the position and orientation of display 114. The generated 2D imagemay be a 2D map of the external scene. The 2D image may reflect adesignated location, such as the geographic area in a selected vicinityof platform 100 (e.g., a 10 km radius surrounding the current locationof platform 100). The 2D image may be continuously updated in accordancewith the changing position and orientation of display 114, to reflectthe changing external scene viewed though display 114 or to reflect anupdated geographic location of platform 100.

Processor 112 further generates (or receives) a 3D image of the externalscene based on the 3D image data (e.g., from real-time images capturedby sensor 118, or from a 3D geographic model or other information storedin database 120) and based on the position and orientation of display114. The 3D image may represent a map image depicting the external sceneviewed along the line-of-sight of display 114. In other words, the 3Dimage depicts the scene from the perspective of a LOS vector calculatedbased on the position and orientation of display 114. For example, ifdisplay 114 is an HMD, the LOS vector may correspond to theline-of-sight of the user (e.g., represented by the head directionand/or eye direction of the user). The sightline of display 114 mayrepresent any direction with respect to platform 100, including aforward-facing view, a rear-facing view, or a side-facing view (i.e., inrelation to the direction of motion of platform 100 and/or display 114).The scene represented by the 3D image at least partially overlaps thescene represented by the 2D image, such that there are at least somecommon image features in both images. The 3D image may be continuouslyupdated in accordance with the changing position and orientation ofdisplay 114, to reflect the changing perspective of the external scene.Processor 112 may also generate the 3D image in advance based onrelevant information, such as based on the location and movementtrajectory of platform 100 or a region of interest. For example,processor 112 may (e.g., repeatedly) obtain the geographic coordinatesof platform 100, as well as the heading, velocity, acceleration,travelling route, and other motion parameters (e.g., using a GPS orINS), determine a future external scene (as viewed along theline-of-sight of display 114) corresponding to an expected futurelocation of platform 100, and then retrieve or generate a 3D imagereflecting the determined future scene.

Display 114 selectively displays the 2D image or the 3D image as anaugmented reality image viewable by the user. The 3D image is displayedoverlaid onto the view of the external scene conforming to the viewpointof the user in accordance with the line-of-sight of display 114. Thedisplayed images may include an indication of at least one region ofinterest (ROI) in the scene. For example, the user may select andindicate an ROI in the scene, which may be tracked by processor 112 anddepicted in the displayed 2D image or 3D image, such as with a visualmarking or symbol. In another example, processor 112 may automaticallydetermine an ROI to be displayed, such as in accordance with thelocation of platform 100 or a designated location (e.g., a specifiedradius around platform 100).

The displayed image is adaptively modified on display 114 to graduallytransition from the 2D image into the 3D image, or vice-versa. A“gradual transition” of displayed images may be considered one in whichat least one intermediate image is displayed in between the initialimage and the final image, where the intermediate image(s) includes(changing or distorted) features or information of the initial and/orfinal image. For example, a gradual transition from a 2D image into a 3Dimage may include one or more intermediate images (or “image frames”)depicting image features of the initial 2D image in a transitionarystate (e.g., having a different size, shape, position, or other visualattributes) as the two-dimensional representation of the external sceneis gradually altered into a three-dimensional form. The transition rate,or time required to transition between the 2D image and 3D image, mayvary and may be modified. In general, display 114 gradually transitionsbetween the images in a manner that allows for the visual tracking oftransitioning image features. Accordingly, the gradual transitioningdisplay provides an intuitive understanding of different scene featuresdepicted in the images in relation to the physical surroundings of theuser. For example, the user may not clearly comprehend the preciselocation in the physical environment of a particular landmark depictedon a 2D map image. However, as display 114 gradually transitions fromdisplaying the 2D map image depicting the marked landmark to displayinga 3D map image depicting the marked landmark, while ensuring the updatedimage reflects the changing external scene viewed though display 114,then the enhanced 3D perspective can enable the user to visualize andunderstand the location of the indicated landmark in relation to thephysical environment. The user may also be able to identify a geographicfeature that is not visible in a 3D map image, such as due to a physicalobstruction that serves to obstruct the feature, as the 3D map imagegradually transitions into a corresponding 2D map image in which thesame feature is no longer obstructed. Display 114 may also depictsupplementary information associated with a ROI in the scene, such as atext describing instructions or details associated with a particularterrain feature, or to indicate objects obstructed from view. The usermay provide instructions to manipulate the displayed images or modifydisplay settings, such as to provide a cross-sectional view through aselected portion of the displayed 2D or 3D image or to change theviewing angle or magnification level of the displayed image.

Reference is now made to FIG. 2A, which is an illustrative view of anexemplary displayed 2D map image, generally referenced 140, withindicated regions of interest, operative in accordance with anembodiment of the present invention. 2D map image 140 represents a mapimage depicting an external environment, such as along the flight pathof an aircraft. 2D map image 140 includes indications of two buildings,referenced 142 and 144, located within a region of interest (ROI)referenced 146. By viewing 2D map image 140, the user may observevarious environmental features around the current location, includingbuildings 142, 144 and ROI 146, but may be unable to accurately relatethe depicted image features to the corresponding real-world features.For example, as buildings 142, 144 are situated relatively close to oneanother, it may be difficult to accurately differentiate between themwhen examined in the physical environment, particularly if buildings142, 144 have similar visual attributes.

At a certain stage, 2D map image 140 begins gradually transitioning intoa corresponding 3D map image on display 114. The displayed transitioningimages may be repeatedly updated to reflect a changing perspective ofthe scene in accordance with changes in the position and orientation ofdisplay 114. The transitioning from 2D to 3D may occur upon instructionsfrom the user, or may occur automatically when certain conditions aremet, such as if the attributes of the 2D image and/or indicated ROIs aresuch that a 3D image would be considered to enhance situationalawareness. Reference is now made to FIG. 2B, which is an illustrativeview of an exemplary displayed 3D map image, generally referenced 150,with indicated regions of interest, corresponding to the 2D map image ofFIG. 2A, operative in accordance with an embodiment of the presentinvention. Display 114 presents 3D map image 150 representing a 3Dperspective image of the external environment as viewed along theline-of-sight of display 114. 3D map image 150 also includes indicationsof two buildings, referenced 152, 154, corresponding to the respectivebuildings 142, 144 indicated in 2D image 140, as well as an ROI 156corresponding to ROI 146 indicated in 2D image 140. As the userperceives 2D map image 140 gradually forming into 3D map image 150 ondisplay 114, the user obtains a perspective view of the externalenvironment that facilitates and enhances user comprehension of thedepicted environmental features in relation to the physicalsurroundings. In particular, the user can track (two-dimensional)buildings 142, 144 gradually transitioning into (three-dimensional)buildings 152, 154, respectively, and thereby distinctly identify theactual corresponding buildings in the external environment. For example,if processor 112 provides instructions or information relating to an ROI146 displayed on the 2D map image 140, the instructions or informationmay be clearly and accurately understood by the user when viewing theindication of ROI 146 on 2D map image 140 transitioning into theindication of ROI 156 on 3D map image 150. The perspectivethree-dimensional visualization of ROI 156 in relation to the physicalenvironment around platform 100 serves to facilitate perception of ROI156 and the understanding of the instructions relating to ROI 156, thusenhancing overall situational awareness. 3D map image 150 maysubsequently transition back into 2D map image 140 on display 114, suchas automatically or upon manual instructions.

An ROI displayed on a 2D image may not be visible when viewed on thecorresponding 3D image, such as if the region or object is obstructedfrom view in the 3D image (from the viewpoint of the LOS of display114). For example, an ROI may be positioned behind a mountain or tree orotherwise blocked by some topographical or terrain feature, or cannot beseen due to atmospheric or weather conditions such as smoke or fog. Theobstructed ROI may also not be visible in the actual physicalenvironment, such as when viewed through a HUD. In such a case, theobstructed ROI may appear and be indicated on the 2D image but wouldgradually vanish as the 2D image gradually transitions into a 3D image,until eventually disappearing entirely. Therefore, if a particular ROIis obstructed or not visible in a 3D image (or in a view of the externalscene viewed through display 114), the 3D image may be transitioned intoa corresponding 2D image, upon manual instructions or upon an automaticdetermination. For example, if the user sees a particular object in aninitial 2D image and then notices that the object is no longer visibleafter then 2D image transitioned into a 3D image, then the user mayprovide instructions to transition back to the 2D image so as to restorethe view of the obstructed object. Optionally, a displayed 2D image mayinclude an indication that a particular ROI would potentially beobstructed when viewed in a corresponding 3D image. For example, apotential 3D obstruction of a 2D image ROI may be indicated by markingthe ROI with a graphical variation with respect to the other regionsdepicted in the scene, such as by displaying the ROI marking with adifferent color and/or different contour line. Correspondingly, anobstructed ROI may also be depicted symbolically in a 3D image, such asby displaying a dashed outline or highlighted shape superimposed ontoanother region of the scene, indicating that the obstructed ROI iseffectively located “behind” the marked scene region (when viewed alongthe LOS of display 114).

Reference is made to FIG. 3, which is a schematic illustration of anexemplary 2D image, generally referenced 160, gradually transitioninginto a 3D image, generally referenced 180, operative in accordance withan embodiment of the present invention. 2D image 160 depicts a scenethat includes at least three scene features, referenced 162, 164 and166, respectively. Scene features 162, 164, 166 appear as 2D objects on2D image 160. As 2D image 160 gradually transitions into a corresponding3D image 180, the scene features 162, 164, 166, gradually appear as 3Dobjects, referenced 182, 184 and 188, respectively, as viewed from aparticular LOS associated with the display. The viewer can thusvisualize the 2D scene features 162, 164, 166 as 3D objects 182, 184 and188 with depth perspective and better perceive the characteristics ofthese objects in relation to the imaged scene, such as theirthree-dimensional shape, size and relative locations. Transitional image170 represents an intermediate stage in the gradual transitioningbetween 2D image 160 and 3D image 180 (or vice-versa). The scenefeatures 162, 164, 166 of 2D image 160 gradually attain athree-dimensional appearance, which is depicted as an intermediate stagein image 170 as corresponding image features 172, 174 and 176, which maybe considered a partial-2D and/or partial-3D phase. The gradualtransitioning from 2D image 160 to 3D image 180, or conversely from 3Dimage 180 to 2D image 160, serves to provide the viewer with anintuitive perception of the depicted scene features in relation to theactual physical environment, while maintaining such intuitive perceptionthroughout the transitioning process.

Different parameters associated with the displayed 2D image or 3D image,and/or the gradual transitioning between the 2D image and 3D image, maybe selectively modified. The display parameters may be selected manuallyvia instructions provided by the user, or automatically, such as viapreconfigured settings. The user may provide default parameters duringan initialization process of system 110, may manually select displayparameters in real-time, and/or may define conditions for altering oradjusting the display parameters automatically. For example, the rate atwhich the 2D image transitions into the 3D image, or vice-versa, may beadjusted or modified, such as by increasing the transition rate toprovide a “quicker” or shorter duration transition (e.g., by decreasingthe number of intermediate image frames displayed between the initialand final images), or alternatively by decreasing the transition rate toprovide a “slower” or longer duration transition (e.g., by increasingthe number of intermediate image frames displayed between the initialand final images). The intermediate image frames may also be displayedrecursively or repeated indefinitely in order (i.e., displayed in a“loop”). Other parameters that may be manipulated may include: thedisplay orientation of the image; the magnification or scaling factor atwhich the image is displayed; the content of the image, such asselecting at least one ROI to be depicted or not depicted in the image;color or brightness parameters or other visual attributes of the image;and the like. The user may also selectively control the degree ofthree-dimensional visualization of the displayed images, by increasingor decreasing the level of conformity of the 3D image overlaid onto theview of display 114.

Image features may remain consistent or may change during the gradualtransitioning between the 2D image and 3D image. For example, certainfeatures or objects depicted in the 2D image may gradually disappear asthe 2D image gradually transitions into a 3D image, until eventuallyonly the borderlines or contours of those features or objects appearvisible. For another example, supplementary image content, such as atext or symbol, present in the 2D image may gradually appear elevatedover the image as it is gradually transitioning into a 3D image.Additionally, supplementary content that is not present in an initial 2Dimage or 3D image may optionally appear during the gradualtransitioning, and conversely, existing supplementary content maygradually disappear during the gradual transitioning.

According to an example implementation of system 110, a driver of avehicle is navigating toward an intended destination. Display 114displays a 2D image of a road map depicting streets and trafficintersections along the driving route of the vehicle, in accordance withthe updated position and orientation of the vehicle. The driver mayreceive instructions (e.g., via a navigation assistance application) toturn right at a particular intersection onto a particular street, suchas by a verbal cue stating: “in another 500 meters, turn right”. Thedriver may not clearly comprehend the instructions, since there may bemultiple adjacent intersections near the indicated area, and thus maynot recognize precisely at which intersection the instructions arereferring to. Even if the particular intersection or street is marked onthe 2D image of the road map, the driver may not understand which of themultiple adjacent intersections or streets it is intended to representin the real-world. Accordingly, display 114 displays the 2D image roadmap gradually transitioning into a 3D image showing the externalenvironment at the updated position and orientation of the vehicle asviewed from the LOS of the see-through display 114 viewed by the driver.In this manner, the driver can track the marked intersection(representing the “turning instructions”) on the 2D image graduallychanging into a three-dimensional perspective view of the intersectionon the 3D image, such that the driver can identify the intersection inthe real-world and is aware of where exactly he needs to implement theturn. For example, the driver may view the transitioning images on asee-through display integrated with the vehicle windshield, or on adisplay screen of a smartphone or other portable computing device, inconjunction with the navigation assistance application.

Reference is now made to FIG. 4, which is a schematic illustration of aviewer observing a 2D image with an indicated ROI graduallytransitioning into a 3D indication of the ROI displayed conformally ontoa view of a real-world scene, operative in accordance with an embodimentof the present invention. A user, represented by eyes 212, views areal-world scene, referenced 202, through a see-through display,represented by visor 214. Scene 202 includes a number of similarbuildings located next to one another on the side of a road. The useralso views a 2D image of a road map, referenced 220, displayed on aportion of visor 214. 2D map image 220 depicts a building of interestmarked with an indication symbol 222. As scene 202 includes multipleadjacent buildings with a similar appearance, the user may not recognizewhich of the real-world buildings is actually represented by symbol 222on 2D image 220. The user then views the 2D indication symbol 222gradually transitioning into a 3D indication symbol 228 overlaidconformally onto the relevant building 204 of scene 202 as viewedthrough the visor display 214. For example, the user views a pluralityof intermediate image frames, represented by intermediate symbols 223,224, 225, 226 and 227, displayed temporally between 2D indication symbol222 and 3D indication symbol 228. The attributes of the displayedtransitioning or intermediate images may be dynamically selected, suchas: the transition resolution or number of intermediate frames, theframe rate, the frame duration, whether to display in a loop(recursively), and the like. By visualizing 2D symbol 222 on 2D image220 gradually transitioning into a 3D symbol 228 overlaid onto the viewof scene 202, the user clearly recognizes that building 204 representsthe indicated building of interest, and can distinctly identify therelevant building 204 in relation to other (similar) buildings in thereal-world environment. It is noted that an indication of an ROI mayrelate to a very precise aspect or subsection of the designated regionor object, such as a specific floor or even a specific room of abuilding of interest, where the degree of potential ambiguity is evengreater, in which case the gradually transitioning intermediate images(or intermediate indication symbols) may enable the user to accuratelyperceive even such precise aspects of the designated region or object.

Reference is now made to FIG. 5, which is a block diagram of a methodfor enhancing the situational awareness of a user, operative inaccordance with an embodiment of the present invention. In procedure252, the position and orientation of a movable see-through display isdetected. Referring to FIG. 1, the position and orientation of display114 is detected by LOS detector 122 and provided to processor 112. Theposition and orientation of display 114 represents a viewpoint of anexternal scene viewed though display 114.

In procedure 254, 2D image data relating to an external scene isreceived. Referring to FIG. 1, processor 112 receives 2D image data ofan external scene, such as real-time images of the scene (e.g., obtainedfrom image sensor 118), and/or previously captured images and/orenvironmental information of the scene (e.g., obtained from database120).

In procedure 256, 3D image data relating to the external scene isreceived. Referring to FIG. 1, processor 112 receives 3D image data ofthe external scene, such as real-time images of the scene (e.g.,obtained from image sensor 118), and/or previously captured imagesand/or environmental information of the scene (e.g., obtained fromdatabase 120). It is noted that the term “3D image data” encompassesgeneral three-dimensional information relating to a geographicenvironment, such as terrain elevation or depth (e.g., obtained from a3D geographic model or digital elevation map), which may not necessarilydirectly reflect an actual image. The 3D image data may also includetextural data relating to geographical terrain features, as well ashierarchical map information, such as embedded maps detailing differentportions or sub-regions of a larger map area, or classifications ofvarious geographic features present at a selected location.

In an optional procedure 260, at least one region of interest (ROI) inthe scene is selected. Referring to FIG. 2A, buildings 142, 144 and ROI146 are indicated on 2D image 140. The ROIs may be selectedautomatically, such as based on predetermined criteria, or may bemanually selected by the user, such as using a cursor or designator toolof user interface 116.

In procedure 262, a 2D augmented reality image of the external scene isdisplayed on the see-through display. It is noted that the 2D image ofthe external scene may include any portion of a particular scene (i.e.,and not necessarily an “entire scene”), such as only selected regions orobjects in the scene. Referring to FIG. 1, display 114 displays anaugmented reality 2D image generated from the received 2D image data.The 2D image may reflect a designated area or location in relation toplatform 100 (e.g., a designated radius). The 2D image may becontinuously updated in accordance with the updated position andorientation of display 114 to reflect the changing external scene, suchas resulting from the changing location of platform 100. The displayed2D image may include an indication of at least one ROI in the scene.Referring to FIG. 2A, display 114 displays 2D map image 140 thatincludes indications of buildings 142, 144 and ROI 146.

In procedure 264, a 3D augmented reality image of the external scene isdisplayed on the see-through display overlaid conformally onto a view ofthe scene. It is noted that the 3D image of the external scene mayinclude any portion of a particular scene (i.e., and not necessarily an“entire scene”) such as only selected regions or objects in the scene.Referring to FIG. 1, display 114 displays an augmented reality 3D imagegenerated from the received 3D image data. The 3D image depicts theexternal scene viewed from the perspective of the LOS of display 114.The scene represented by the 3D image at least partially overlaps thescene represented by the 2D image, such that there are at least somecommon image features in both the 2D image and the 3D image. The 3Dimage may be continuously updated in accordance with the changingposition and orientation of display 114, to reflect the changingperspective of the external scene. Processor 112 may also generate the3D image in advance based on predefined information, such as thelocation and movement of platform 100 or a region of interest for theuser, such as in accordance with a selected radius surrounding thecurrent location of platform 100 (corresponding to the updated positionand orientation of display 114), or a heading (directional) vector ormotion trajectory of platform 100 (or display 114).

The displayed 3D image may include an indication of at least one ROI inthe scene. Referring to FIG. 2B, display 114 displays 3D map image 150including indications of buildings 152, 154 and ROI 156.

In procedure 266, a gradual transition of the 2D image into the 3D imageis displayed on the see-through display. Referring to FIG. 1, display114 displays a gradual transition of the 2D image into the 3D image.Display 114 displays at least one intermediate image frame depictingimage features of the 2D image in a transitionary state (e.g., having adifferent size, shape, position, or other visual attributes) as thetwo-dimensional representation of the external scene is graduallyaltered into a three-dimensional form. The transition is implemented ina manner that allows visual tracking of transitioning image features.The transition rate or duration of transitioning may be selectivelymodified. The image transition may be initiated upon manual instructionsor automatically upon predetermined conditions. Referring to FIGS. 2Aand 2B, display 114 displays 2D map image 140 gradually transitioninginto 3D map image 150. The ROI indications are maintained during theimage transitioning, where for example, ROI 146 on 2D map image 140 isgradually altered into corresponding ROI 156 on 3D map image 150. Otherfeatures of 2D map image 140 may also gradually alter to attain aperspective three-dimensional appearance in 3D map image 150, providinga perspective view of the environment and facilitating identification ofthe depicted features. For example, the user can track ROIs 142, 144gradually transitioning into respective ROIs 152, 154, providingenhanced comprehension of the exact locations of buildings 152, 154 inrelation to the actual physical environment. It is noted that thegradual transition into a 3D image may also encompass athree-dimensional view of the external scene as viewed through thesee-through display 114, i.e., rather than a separate “image”, such thatthe displayed 2D image gradually transitions into a 3D view of theexternal scene (from the perspective of the viewer).

In procedure 268, a gradual transition of the 3D image into the 2D imageis displayed on the see-through display. Referring to FIG. 1, display114 displays a gradual transition of the 3D image into the 2D image.Display 114 displays at least one intermediate image frame depictingimage features of the 3D image in a transitionary state (e.g., having adifferent size, shape, position, or other visual attributes) as thethree-dimensional representation of the external scene is graduallyaltered into a two-dimensional form. The transition is implemented in amanner that allows visual tracking of transitioning image features. Thetransition rate or duration of transitioning may be selectivelymodified. The image transition may be initiated upon manual instructionsor automatically upon predetermined conditions. Referring to FIGS. 2Aand 2B, display 114 displays 3D map image 150 gradually transitioninginto corresponding 2D map image 140. The indications of respectivebuildings 152, 154 and ROI 156 of 3D image 150 are gradually alteredinto indications of respective buildings 142, 144 and ROI 146 in 2Dimage 140. Other image features may also be gradually altered from aperspective three-dimensional appearance of 3D image 150 into anon-perspective two-dimensional appearance in 2D image 140. The 2D imagemay depict environmental features that are not visible or obstructedfrom view in the 3D image, allowing the user to clearly perceive suchobstructed features and their attributes and location in relation to theactual physical environment.

The method of FIG. 5 is generally implemented in an iterative manner,such that at least some of the procedures are performed repeatedly, inorder to maintain the display of a 2D image or a 3D image or a gradualtransition between the 2D image and 3D image, of an external scenereflecting the location of the system or platform over a sequence ofimage frames (i.e., such that the displayed images are linked to thechanging location of the platform for at least a selected duration).

While certain embodiments of the disclosed subject matter have beendescribed, so as to enable one of skill in the art to practice thepresent invention, the preceding description is intended to be exemplaryonly. It should not be used to limit the scope of the disclosed subjectmatter, which should be determined by reference to the following claims.

The invention claimed is:
 1. A system for enhancing the situationalawareness of a user, the system comprising: a movable see-throughdisplay viewable by the user, and configured to display a syntheticimage while providing a see-through view of a real-world external scenethrough the display; and a processor, configured to receive: (i) anupdated position and orientation of the see-through display; (ii)two-dimensional (2D) image data relating to the external scene; and(iii) three-dimensional (3D) image data relating to the external scene,wherein the see-through display is configured to display an augmentedreality 2D image of the real-world external scene based on the received2D image data, in accordance with the updated position and orientationof the see-through display, while providing a see-through view of thereal-world external scene, the 2D image gradually transitioning into anaugmented reality 3D image of the external scene based on the received3D image data, the 3D image overlaid conformally onto the see-throughview of the real-world external scene viewed by the user through thesee-through display, in accordance with the updated position andorientation of the see-through display; and wherein the see-throughdisplay is further configured to display an augmented reality 3D imageof the external scene based on the received 3D image data, the 3D imageoverlaid conformally onto the see-through view of the real-worldexternal scene viewed by the user through the see-through display, inaccordance with the updated position and orientation of the see-throughdisplay, the 3D image gradually transitioning into an augmented reality2D image of the real-world external scene based on the received 2D imagedata, in accordance with the updated position and orientation of thesee-through display, while maintaining a see-through view of thereal-world external scene.
 2. The system of claim 1, wherein at leastone of the 2D image and the 3D image comprises at least one region ofinterest (ROI) selected from the group consisting of: an ROI selectedbased on the updated position and orientation of the see-throughdisplay; and an ROI selected by the user.
 3. The system of claim 1,wherein the 3D image depicts at least one region of interest (ROI)obstructed by features in the external scene.
 4. The system of claim 1,wherein at least one image feature of at least one of the 2D image andthe 3D image gradually disappears during the gradual transition.
 5. Thesystem of claim 1, wherein at least one image feature of at least one ofthe 2D image and the 3D image gradually appears during the gradualtransition.
 6. The system of claim 1, wherein the display is furtherconfigured to display supplementary image content during the gradualtransition.
 7. The system of claim 1, wherein at least one transitioningattribute of the gradual transition is selectively modified.
 8. Thesystem of claim 1, wherein the see-through display is configured torepeatedly update the gradual transition in accordance with the updatedposition and orientation of the see-through display.
 9. The system ofclaim 1, wherein the 3D image is generated in advance based on at leastone factor selected from the group consisting of: a radius surroundingthe updated position and orientation of the see-through display; aheading vector of the see-through display; a selected region of interest(ROI); and predefined information.
 10. The system of claim 1, whereinthe movable see-through display comprises a display of a movingplatform.
 11. The system of claim 10, wherein the moving platformcomprises an aircraft in flight.
 12. A method for enhancing thesituational awareness of a user, the method comprising the proceduresof: receiving an updated position and orientation of a movablesee-through display viewable by the user, and configured to display asynthetic image while providing a see-through view of a real-worldexternal scene through the display; receiving 2D image data relating tothe external scene; receiving 3D image data relating to the externalscene; displaying on the see-through display an augmented reality 2Dimage of the real-world external scene based on the received 2D imagedata, in accordance with the updated position and orientation of thesee-through display, while providing a see-through view of thereal-world external scene, the 2D image gradually transitioning into anaugmented reality 3D image of the external scene based on the received3D image data, the 3D image overlaid conformally onto the see-throughview of the real-world external scene viewed by the user through thesee-through display, in accordance with the updated position andorientation of the see-through display; and displaying on thesee-through display an augmented reality 3D image of the external scenebased on the received 3D image data, the 3D image overlaid conformallyonto the see-through view of the real-world external scene viewed by theuser through the see-through display, in accordance with the updatedposition and orientation of the see-through display, the 3D imagegradually transitioning into an augmented reality 2D image of thereal-world external scene based on the received 2D image data, inaccordance with the updated position and orientation of the see-throughdisplay, while maintaining a see-through view of the real-world externalscene.
 13. The method of claim 12, wherein at least one of the 2D imageand the 3D image comprises at least one region of interest (ROI)selected from the group consisting of: an ROI selected based on theupdated position and orientation of the see-through display; and an ROIselected by the user.
 14. The method of claim 12, wherein the 3D imagedepicts at least one region of interest (ROI) obstructed by features inthe external scene.
 15. The method of claim 12, wherein at least oneimage feature of at least one of the 2D image and the 3D image graduallydisappears during the gradual transition.
 16. The method of claim 12,wherein at least one image feature of at least one of the 2D image andthe 3D image gradually appears during the gradual transition.
 17. Themethod of claim 12, wherein at least one transitioning attribute of thegradual transition is selectively modified.
 18. The method of claim 12,wherein the gradual transition is repeatedly updated in accordance withthe updated position and orientation of the see-through display.
 19. Themethod of claim 12, wherein the 3D image is generated in advance basedon at least one factor selected from the group consisting of: a radiussurrounding the updated position and orientation of the see-throughdisplay; a heading vector of the see-through display; a selected regionof interest (ROI); and predefined information.